Ranking User-annotated Images for Multiple Query Terms

This paper examines the task of searching photographs shared on the Flickr website (http://www.flickr.com/) to find images which contain objects matching user queries. If, for example, the user enters the search terms ‘cat’ and ‘dog’, we can quickly find potentially-relevant images using the textual tags associated with the images. Our aim is to rank those potentially-relevant images and return to the user the images which we are most confident do in fact contain a cat and a dog. The system identifies what is consistent across images to learn the visual meaning of user queries and return relevant images to the user. We improve on previous approaches to this task by taking into account the relationships between photographs and the users who uploaded them, to avoid being overly influenced by individual users’ inaccurate or idiosyncratic use of textual tags. We focus on user searches containing multiple terms, and show that image ranking performance for multiple query terms can be improved by learning a combined model for the query and adding images which match only a subset of the query terms as negative training examples. Search queries are answered by using a mixture of kernel density estimators to rank the visual content of web images from the Flickr website whose noisy tag annotations match the given query terms. We estimate the probability that each image in the query set was generated from the query-related class: